Connect with us

AR/VR

Build a Streaming Application with Face Filter on Android

Avatar

Published

on

Shaocheng Yang
  1. A basic-to-intermediate understanding of Java and Android SDK
  2. Agora.io account
  3. Banuba Face AR demo app
  4. Android Studio and 2 Android devices

This guide will go over the steps for building a live streaming application on Android using the Agora SDK. This is a list of the core features that will be included in our app:

  • Users can set up virtual rooms to host live streams and become a streamer.
  • Users can find all live streams and join the virtual room as an audience.
  • Streamers can use the face filter feature to stream with a virtual mask or animation.
  • Streamers can change their voice through a voice changer.
  • Audiences in a virtual room can send text messages which can be seen by everyone in that room.
  • Users can search for other users by their name and send private text messages to them.

It is easier to build our application on top of the Banuba FaceAR demo apps. The package contains a default Banuba SDK Demo app demonstrating major Banuba SDK features and a beautification example app with neural networks coloring feature. We are going to focus on the Banuba SDK Demo app for our project. Feel free to explore the beautification example app on your own.

https://docs.banuba.com/docs/android/android_demo_app_ui

1. How to use subtle AR filters to survive your Zoom meetings?

2. The First No-Headset Virtual Monitor

3. Augmented reality (AR) is the future of Restaurant Menu?

4. Creating remote MR productions

To integrate Agora SDK in your project, you need to add the following lines in the /app/build.gradle file of the project.

//Agora RTC SDK for video call
implementation 'io.agora.rtc:full-sdk:3.0.1.1'//Agora RTM SDK for chat messaging
implementation 'io.agora.rtm:rtm-sdk:1.2.2'
//Agora UIKit
implementation 'io.agora.uikit:agorauikit:2.0.1'

Add Project Permissions

The next thing we need to do is to add project permissions in the /app/src/main/AndroidManifest.xml file for device access according to your needs:

<manifest xmlns:android="http://schemas.android.com/apk/res/android"
...
>
<uses-permission android:name="android.permission.READ_PHONE_STATE" />
<uses-permission android:name="android.permission.INTERNET" />
<uses-permission android:name="android.permission.RECORD_AUDIO" />
<uses-permission android:name="android.permission.CAMERA" />
<uses-permission android:name="android.permission.MODIFY_AUDIO_SETTINGS" />
<uses-permission android:name="android.permission.ACCESS_NETWORK_STATE" />
<!-- The Agora SDK requires Bluetooth permissions in case users are using Bluetooth devices. -->
<uses-permission android:name="android.permission.BLUETOOTH" />
<uses-permission android:name="android.permission.ACCESS_WIFI_STATE" />
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />
...
</manifest>

Since our application allows users to search and chat with other users in the app, we need to use Firebase Realtime Database to save users’ account information. Here are the steps you need to connect your application to Firebase Realtime Database:

First, let’s build our application from the first page. Same as most of the streaming applications, our application needs a user login page. And that should look like this:

public void onLoginNextClick(View view) {
EditText userNameEditText = findViewById(R.id.et_login_user_name);
String userName = userNameEditText.getText().toString();

if(userName == null || userName.equals("")) {
Toast.makeText(this, "user name cannot be empty", Toast.LENGTH_SHORT).show();
}else {
Intent intent = new Intent(this, HomeActivity.class);
intent.putExtra("userName", userName);
startActivity(intent);
}
}

In the HomeActivity, we will create a three-page bottom navigation bar with a fragment connected to each page. If the user taps on the first bottom navigation menu, he will be directed to the Live fragment. Similarly, the second menu will direct the user to the Chat fragment and the last one is the Other fragment.

@Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_home);
userName = getIntent().getStringExtra("userName");

saveUserOnFireDatabase(userName);
}

private void saveUserOnFireDatabase(String userName) {
mRef = FirebaseDatabase.getInstance().getReference("Users");
mRef.push();
mRef.child(userName).setValue(new DBUser(userName, false));
}

Live Fragment

The Live fragment is to show all the live streamers’ names and allow other users to join the streamer’s virtual room by clicking a button.

childEventListener = new ChildEventListener() {
@Override
public void onChildAdded(@NonNull DataSnapshot dataSnapshot, @Nullable String s) {
DBUser result = dataSnapshot.getValue(DBUser.class);
if (!result.getName().equals(userName) && result.getStreaming() == true) {
streamerList.add(result);
}
}

...
};

mRef.orderByChild("name").addChildEventListener(childEventListener);

private void joinStream(String streamerName) {
Intent intent = new Intent(getActivity(), AudienceActivity.class);
intent.putExtra("streamerName", streamerName);
intent.putExtra("userName", userName);
startActivity(intent);
}
@Override
public void onClick(View v) {
Intent intent = new Intent(getActivity(), MainActivity.class);
intent.putExtra("userName", userName);
startActivity(intent);
}

Chat Fragment

The Chat fragment is the place for users to search for other users and send private text messages to them. So you will build your UI like this:

Let’s open the MainActivity class. From the previous steps, we know this activity is for adding the face filter features on the local camera views. Our job here is to send these local camera views with face filters out to the remote users.

Initialize RtcEngine and Join Channel

Inside the onCreate callback, get the username passed from the home activity and initialize the Agora RtcEngine.

userName = getIntent().getStringExtra("userName");

try {
mRtcEngine = RtcEngine.create(getBaseContext(), getString(appID), mRtcEventHandler);
} catch (Exception e) {
Log.e(TAG, Log.getStackTraceString(e));
throw new RuntimeException("NEED TO check rtc sdk init fatal errorn" + Log.getStackTraceString(e));
}

mRtcEngine.enableVideo();

  1. Click the Project Management tab on the left navigation panel.
  2. Click “Create” and follow the on-screen instructions to set the project name, choose an authentication mechanism, and click “Submit”.
  3. On the Project Management page, find the App ID of your project.
mRtcEngine.setExternalVideoSource(true, false, true);
mRtcEngine.joinChannel(token, channelName, "Extra Info", 0);

Push External Video Frame

We have already let the Agora RtcEngine know that we will use the external video source as the video input. However, we haven’t provided the video frame yet.

mSdkManager.startForwardingFrames();
@Override
public void onFrameRendered(@NonNull Data data, int width, int height) {
byte[] arr = new byte[data.data.remaining()];
data.data.get(arr);

AgoraVideoFrame agoraVideoFrame = new AgoraVideoFrame();
agoraVideoFrame.buf = arr;
agoraVideoFrame.stride = width;
agoraVideoFrame.height = height;
agoraVideoFrame.format = AgoraVideoFrame.FORMAT_RGBA;
agoraVideoFrame.timeStamp = System.currentTimeMillis();
mRtcEngine.pushExternalVideoFrame(agoraVideoFrame);
data.close();
}

Add the Voice Changer

To make the streaming experience more interesting, we can add a voice changer in the application. Create a button in the MainActivity and set the onClick method for that. When the user clicks that, we will call setLocalVoiceChanger to set the voice changer.

public void onVoiceChangerClick(View view) {
mRtcEngine.setLocalVoiceChanger(VOICE_CHANGER_BABYGIRL);
Toast.makeText(this, "Voice Changer ON", Toast.LENGTH_SHORT).show();
}

Setting up the audience view is easier. Let’s go to the Audience activity and use the Agora UIKit.

@Override
protected void onCreate(Bundle savedInstanceState) {
...
AgoraRTC.instance().bootstrap(this, appID, channelName);
setContentView(R.layout.group_template);
}
AgoraRTC.instance().muteLocalVideoStream(true);
AgoraRTC.instance().muteLocalAudioStream(true);
IRtcEngineEventHandler eventHandler = new IRtcEngineEventHandler() {

@Override
public void onRemoteVideoStateChanged(final int uid, int state, int reason, int elapsed) {
super.onRemoteVideoStateChanged(uid, state, reason, elapsed);

if (state == REMOTE_VIDEO_STATE_STARTING) {
runOnUiThread(new Runnable() {
@Override
public void run() {
agoraView.setUID(uid);
}
});
}
}
};

@Override
protected void onCreate(Bundle savedInstanceState) {
...
agoraView = findViewById(R.id.max_view);
AgoraRTC.instance().registerListener(eventHandler);
...
}
endCallButton.setOnClickListener(new View.OnClickListener() {
@Override
public void onClick(View v) {
finish();
}
});

Text messaging in a streaming application can be divided into two parts: channel messaging and private messaging. The first one sends messages to the channel and everyone in the channel can receive that message while the second one sends private messages from a peer to another peer.

Channel Messaging

We want to allow all the audiences to send messages in the streamer’s room. To achieve that, we need to create a RtmClient using Agora Messaging SDK.

mRtmclient = RtmClient.createInstance(mContext, appID, new RtmClientListener() {
@Override
public void onConnectionStateChanged(int state, int reason){
for (RtmClientListener listener : mListenerList) {
listener.onConnectionStateChanged(state, reason);
}
}@Override
public void onMessageReceived(RtmMessage rtmMessage, String peerId) { ... }
});
mRtmClient.login(null, userName, new io.agora.rtm.ResultCallback<Void>() {
@Override
public void onSuccess(Void aVoid) {
...
}
@Override
public void onFailure(ErrorInfo errorInfo) {
...
}
});
mRtmChannel = mRtmClient.createChannel(mChannelName, new MyChannelListener());
class MyChannelListener implements RtmChannelListener {
@Override
public void onMessageReceived(final RtmMessage message, final RtmChannelMember fromMember) {
runOnUiThread(new Runnable() {
@Override
public void run() {
String account = fromMember.getUserId();
String msg = message.getText();
MessageBean messageBean = new MessageBean(account, msg, false);
messageBean.setBackground(getMessageColor(account));
mMessageBeanList.add(messageBean);
mMessageAdapter.notifyItemRangeChanged(mMessageBeanList.size(), 1);
mRecyclerView.scrollToPosition(mMessageBeanList.size() - 1);
}
});
}
...
}
mRtmChannel.join(new ResultCallback<Void>() {
@Override
public void onSuccess(Void responseInfo) { ... }@Override
public void onFailure(ErrorInfo errorInfo) { ... }
});
private void sendChannelMessage(String content) {
// step 1: create a message
RtmMessage message = mRtmClient.createMessage();
message.setText(content);
// step 2: send message to channel
mRtmChannel.sendMessage(message, new ResultCallback<Void>() {
@Override
public void onSuccess(Void aVoid) {
...
}

@Override
public void onFailure(ErrorInfo errorInfo) {
...
}
});
}

Private Messaging

In the ChatFragment, users can find their friend by searching the friend’s name and send private text messages to them.

childEventListener = new ChildEventListener() {
@Override
public void onChildAdded(@NonNull DataSnapshot dataSnapshot, @Nullable String s) {
DBUser result = dataSnapshot.getValue(DBUser.class);
showFriendText.setText(result.getName());
mRef.orderByChild("name").startAt(friendName).endAt(friendName + "uf8ff").removeEventListener(childEventListener);
}
...
};

mRef.orderByChild("name").startAt(friendName).endAt(friendName + "uf8ff").addChildEventListener(childEventListener);

  1. You need to register a RtmClientListener and receive messages from the onMessageReceived callback.
class MyRtmClientListener implements RtmClientListener {

@Override
public void onMessageReceived(final RtmMessage message, final String peerId) {
runOnUiThread(new Runnable() {
@Override
public void run() {
String content = message.getText();
if (peerId.equals(mPeerId)) {
MessageBean messageBean = new MessageBean(peerId, content,false);
messageBean.setBackground(getMessageColor(peerId));
mMessageBeanList.add(messageBean);
mMessageAdapter.notifyItemRangeChanged(mMessageBeanList.size(), 1);
mRecyclerView.scrollToPosition(mMessageBeanList.size() - 1);
} else {
MessageUtil.addMessageBean(peerId, content);
}
}
});
}

...
}

mRtmClient.sendMessageToPeer(mPeerId, message, mChatManager.getSendMessageOptions(), new ResultCallback<Void>() {
@Override
public void onSuccess(Void aVoid) {
...
}

@Override
public void onFailure(ErrorInfo errorInfo) {
...
}
});

Now let’s run our application!

Congratulations! You just built yourself a live streaming application with face filter features and chat messaging!

Source: https://arvrjourney.com/build-a-streaming-application-with-face-filter-on-android-9399a028bb40?source=rss—-d01820283d6d—4

AR/VR

Unity announces plans to release experimental version of Unity OpenXR early next year

Avatar

Published

on

In Augmented Reality, Virtual Reality and Mixed Reality News

October 24, 2020 – Unity Technologies, a leading platform for creating and operating interactive, real-time 3D content, has recently announced that it plans to have early previews of its support of OpenXR on some platforms as early as the end of this year (Unity 2020 release cycle). The company noted that it has been closely monitoring the development of the OpenXR standard, and has accelerated its efforts to enable OpenXR in Unity.

OpenXR is an open standard created by the Khronos Group, aimed at simplifying augmented and virtual reality (AR/VR) development by allowing developers to seamlessly target a wide range of AR/VR devices. Unity is a member of the Khronos Group and is an active contributor to the OpenXR standard.

In a blog post on the announcement, Matt Fuad, Sr. Technical Product Manager, AR/VR at Unity, commented: “We’re at an inflection point now, where OpenXR 1.0 has been ratified and OpenXR runtimes by various vendors are reaching maturity. This inflection point has accelerated our efforts to enable OpenXR in Unity. Though we’re still working through some details, we want to reiterate our commitment to OpenXR and are excited to share our latest plans.” 

As OpenXR becomes the primary backend for many platforms, Unity added that it is working with its partners to ensure full Unity support of their OpenXR runtimes, along with the necessary OpenXR extensions to ensure parity with existing features. The company stated that its development efforts and support will be focused on providing the best developer experience on Unity supported platforms.

Furthermore, by enabling OpenXR support on partner platforms, it affords Unity the ability to make support widely available for other OpenXR runtimes/devices. Early next year, the company plans to release an experimental version of Unity OpenXR that works with other conformant OpenXR runtimes based on the OpenXR 1.0 specification. Of course, the company clarified that given the unbounded combinations of possible hardware/software configurations, it will not be able to test or guarantee that all configurations will work optimally. However, as issues are discovered with runtimes, Unity stated that it will work to contribute conformance tests and specification changes back to the Khronos working group to help the community as a whole. Additionally, the company will also make sure that it is clear to developers which platforms have been fully tested and thus supported by Unity.

Fuad added: “Our mission is to make it as easy as possible for developers to take their content to the widest set of desired platforms. OpenXR is an important part of that story, and will be implemented as a part of our XR plug-in framework (XR SDK).”

This means developers will be able to continue using Unity’s suite of XR workflows and frameworks (AR Foundation, XR Interaction Toolkit, Unity MARS) for platforms that adopt OpenXR, as well as platforms that may choose not to adopt it.

Unity noted that it will be announcing more updates over the coming months. For more information on OpenXR, please visit the Khronos Group’s website.

Image credit: Unity Technologies / Khronos Group

About the author

Sam Sprigg

Sam is the Founder and Managing Editor of Auganix. With a background in research and report writing, he covers news articles on both the AR and VR industries. He also has an interest in human augmentation technology as a whole, and does not just limit his learning specifically to the visual experience side of things.

Source: https://www.auganix.org/unity-announces-plans-to-release-experimental-version-of-unity-openxr-early-next-year/

Continue Reading

AR/VR

Competition: Win Either Angry Birds VR or Acron: Attack of the Squirrels! for Oculus Quest

Avatar

Published

on

It’s coming up to the season of giving and what better way to start the celebrations early than with a competition where you can win some videogames. Today’s giveaway is a double bill, with Resolution Games’ Angry Birds VR: Isle of Pigs and Acron: Attack of the Squirrels! up for grabs.

Angry Birds VR

Rovio’s iconic Angry Birds franchise came to VR in 2019 with a selection of core levels which have been expanded upon over time. One of the major updates to the bird flinging experience was the addition of a level builder late last year, followed by online sharing to massively enhance the gameplay possibilities.

Acron: Attack of the Squirrels! on the other hand, is a purely multiplayer experience. One person is in VR controlling a tree trying to protect its golden acorns from a bunch of thieving squirrels. All the squirrels are controlled by players on mobile devices – via a free app – who are able to select characters with particular traits such as burrowing short distances underground. All making for a fast and entertaining party game.

Most recently, Resolution Games released a Halloween update for both titles, adding new levels as well as spooky themes to get into the spirit of the season as a lot of developers like to do.

Acron: Attack of the Squirrels

So onto the competition. VRFocus has got six codes to give away, three for Angry Birds VR: Isle of Pigs and three for Acron: Attack of the Squirrels, both for the Oculus Quest platform. There are multiple ways to enter the giveaway with the standard prize draw entry rules applying: Follow us (or already be following us) on Twitter or alternatively, visit our Facebook page or YouTube channel to get an entry for each. Winners will receive a single randomly drawn code. The competition will be open until 11.59 pm GMT on Monday 26th October 2020. The draw will be made shortly thereafter.

Win Angry Birds VR or Acron: Attack of the Squirrels!

Source: https://www.vrfocus.com/2020/10/competition-win-either-angry-birds-vr-or-acron-attack-of-the-squirrels-for-oculus-quest/

Continue Reading

AR/VR

Sayduck enters three-year license agreement with Forms+Surfaces to create 3D product models for AR visualization

Avatar

Published

on

In Augmented Reality News 

October 23, 2020 – Bublar Group has announced that its subsidiary Sayduck has expanded its collaboration with one of its oldest customers and entered into a three-year licence agreement with design and manufacturing company Forms+Surfaces based in the US.

The new assignment will consist of Sayduck building an online 3D configurator for over 80 of Forms+Surfaces’ different furnishing products for public spaces. Forms+Surfaces designs and manufactures architectural products used in public spaces, from walls and elevator interiors to site furnishings and lighting.

According to Bublar, the 3D configurator will make it possible to realistically present complex products that come in a multitude of variations and combinations that typically are not easily showcased online. It will also allow Forms+Surfaces customers to visualize product selections in augmented reality (AR), a core feature of the Sayduck Platform.

“We are truly excited about our continued collaboration with Sayduck. We knew that we were only scratching the surface as far as business opportunities when Forms+Surfaces introduced the AR technology to architects and designers for select products back in 2013. This new 3D configurator project is a natural extension of our digital outreach, helping streamline the specification process of our entire outdoor product line for our customers,” commented Jeffrey Stork, President, Forms+Surfaces, Inc.

Examples of Forms+Surfaces’ products

Earlier assignments from Forms+Surfaces have included an AR-enabled iOS app. In addition, Sayduck states that it has created multiple products as simpler, non-configurable items in the past.

Commenting on the agreement, Niklas Slotte, Managing Director at Sayduck, said: “I am very happy that we have gained continued trust from one of our oldest customers. We have had a strong collaboration for more than seven years and with this agreement we will develop our partnership even further.”

Bublar added that Sayduck’s strategy is to help product manufacturers and e-commerce companies to showcase and present their products in more immersive ways. The Sayduck Platform allows companies to create unique web 3D and AR experiences for their customers, helping to drive product engagement and customer confidence.

For more information on Sayduck and its 3D Configurator product, please visit the company’s website.

Image credit: Sayduck / Forms+Surfaces

About the author

Sam Sprigg

Sam is the Founder and Managing Editor of Auganix. With a background in research and report writing, he covers news articles on both the AR and VR industries. He also has an interest in human augmentation technology as a whole, and does not just limit his learning specifically to the visual experience side of things.

Source: https://www.auganix.org/sayduck-enters-three-year-license-agreement-with-formssurfaces-to-create-3d-product-models-for-ar-visualization/

Continue Reading
Blockchain News1 hour ago

Concerns Arise as North Korea’s Financial Services Commission Unsure of Its Cryptocurrency Mandate

Esports3 hours ago

G2 close in on NiKo deal

Denmark
Esports4 hours ago

Lyngby Vikings replace ENCE in Elisa Invitational; new groups and schedule revealed

Esports4 hours ago

Madden 2 ‘In Order to Access The Online Features:’ How to Fix the Bug

Esports4 hours ago

Apex Legends Switch Delayed Again

Esports4 hours ago

Pokémon GO Colorful Pokémon Are Revealed

CIS
Esports5 hours ago

Virtus.pro extend unbeaten streak with win over NAVI to reach IEM New York CIS grand final

Denmark
Esports6 hours ago

Heroic eliminate NiP; to face Astralis in DreamHack Open Fall consolidation final

Blockchain News9 hours ago

Kik Survives Legal Battle With the SEC, Kin Crypto to Continue Trading on Exchanges

Esports9 hours ago

How to Play With Friends Online in Dynamax Adventures in Pokémon Sword and Shield The Crown Tundra

Esports10 hours ago

How to Separate and Rejoin Calyrex from Glastrier or Spectrier in Pokémon Sword and Shield Crown Tundra

Energy10 hours ago

Grand Opening of the 25th China (Guzhen) International Lighting Fair

Esports10 hours ago

All increased spawns, encounters, and event raids for the Pokémon Go Halloween Event

Esports10 hours ago

All Timed Research tasks and rewards in Pokémon Go’s 2020 Halloween event

Esports11 hours ago

All ‘A Spooky Message Unmasked’ Special Research tasks and rewards in Pokémon Go

AR/VR12 hours ago

Unity announces plans to release experimental version of Unity OpenXR early next year

Energy14 hours ago

World’s Sole Owner of PERKLONE and TRIKLONE Now Supplying Saudi Arabia Oil & Petroleum Refineries; PERKLONE EXT; Catalyst grade, UOP Approved

Esports15 hours ago

Complexity Gaming Signs Jks From 100 Thieves

Blockchain News18 hours ago

People’s Bank of China Seeks to Prohibit the Creation of Digital Tokens

Energy19 hours ago

EV Charge Holding Limited sichert sich 15 Millionen Euro von GEM

Energy19 hours ago

Xinhua Silk Road: Chinas Solarindustrie soll angesichts der angestrebten Klimaneutralität schneller wachsen

Energy20 hours ago

Xinhua Silk Road: o setor solar da China deve concretizar um crescimento mais rápido em meio a um objetivo de neutralidade de carbono

Energy20 hours ago

Xinhua Silk Road: La industria solar de China adoptará un crecimiento más rápido en busca del objetivo de la neutralidad de carbono

Fintech21 hours ago

HSBC leans into APIs to expand Cash Flow Forecasting tool

Blockchain22 hours ago

U.S. crypto exchanges have a plan for the travel rule. Now they just have to get along

Esports22 hours ago

jks: “I’m at the point in my career where I really just want to win a lot of things; this is the reason why I came to Complexity”

Energy22 hours ago

Insights on the Fermented Ingredients Global Market to 2027 – Strategic Recommendations for New Entrants

Blockchain22 hours ago

Exploring the current state of cryptocurrency taxation in the European Union

Fintech22 hours ago

Canada Partners with Swiss, U.S. Fintechs; Neobank Raises Millions in Mexico

Energy22 hours ago

$6.9 Billion Worldwide Spandex Industry to 2027 – Impact of COVID-19 on the Market

Energy23 hours ago

Kimbell Royalty Partners Declares Third Quarter 2020 Distribution

Denmark
Esports23 hours ago

Vitality edge out Astralis to reach DreamHack Open Fall grand final

Energy23 hours ago

Turquoise Hill completes 10:1 share consolidation

Energy23 hours ago

Insights on the Rayon Fibers Global Market to 2027 – Featuring Grasim Industries, Kelheim Fibres and Lenzing Among Others

Aerospace23 hours ago

Boston startups expand region’s venture capital footprint

Esports23 hours ago

How to Make Excellent Throws in Pokémon GO

Esports23 hours ago

Professor Willow Spotted a Colorful Pokémon: Galarian Ponyta Teased in Pokémon GO

Esports23 hours ago

Adventure Together Pokémon GO: How to Make a Pokémon Your Buddy

Energy23 hours ago

Valvoline to Participate in Gabelli Annual Automotive Aftermarket Symposium on Nov. 3

Energy23 hours ago

$172 Million Worldwide Friction Stir Welding Equipment Industry to 2027 – Impact of COVID-19 on the Market

Trending